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Abstract 

Current planning algorithms have difficulty handling the com- 
plexity that is due to an increase in domain uncertainty, and 
especially in the case of multi-dimensional continuous spaces. 
Therefore, they produce plans that do not take into account nu- 
merous situations that can occur at runtime, such as faults or 
other changes in the planning domain itself. Thus there is a 
gap between the plan generation and the reality experienced 
at runtime. Here we present two methods that allow the plan 
conditionals to be revised w.r.t. uncertainty on the system as 
estimated at runtime. 

Introduction 

The need for autonomy and robustness in the face of uncer- 
tainty is growing as planetary rovers become more capable 
and as missions explore more distant planets. Recent progress 
in areas such as instrument placement (Pedersen et al 2003; 
2005) makes it possible to visit multiple rocks in a single com- 
munication cycle. This requires reasoning over much longer 
time frames, in more uncertain environments. Simple uncon- 
ditional plans as used by the Mars Exploration Rovers (MER) 
will probably have a low probability of success in such con- 
text, so that the robot would spend almost all its time waiting 
for new orders from home. 

In the last decade, architectures for future planetary rover 
missions include a planner/scheduler, a health monitoring sys- 
tem, and an executive. The planner/scheduler generates a con- 
trol program/plan that describes the sequence of run-time ac- 
tions necessary to achieve mission goals. Since the rover’s en- 
vironment is highly uncertain (Bresina etal 2002), the control 
programs (also called plans) are contingency plans (Dearden 
et al 2003) in that they involve conditional branches that are 
based on decision functions of the system state that the execu- 
tive can evaluate in real time. The executive is responsible for 
the execution of the control programs, taking into account the 
current state of the system as estimated by the health monitor- 
ing system. This capability includes deciding the best branch 
in a plan when reaching a branch point, given an estimate of 
the current system state, inserting and replacing plan portions 
to react to faults and other unpredictable events. 

However, planners have difficulties handling certain situa- 
tions, such as actions that carry no utility (typically used for 
responding to unlikely situations) and fault occurences, or to 


prepare for a belief state update 1 . First, actions with no re- 
ward can possibly be inserted anywhere in the plan at low cost, 
so the greedy approach that seeks to maximize the expected 
utility fails to position them efficiently. Second, planner do- 
mains describe a very limited set of faults, thus relying on a 
mostly nominal model of the world and system actions (e.g. 
no stuck wheels, broken navigation system, rocky environ- 
ment,...). Moreover, fault models exponentially increase the 
complexity of the planning even if the faults have low prob- 
ability of occurence as they can occur at any time during the 
plan execution. Finally, the health monitoring system returns 
an ever changing belief state over time that has to be taken into 
account. For these reasons, the response to unlikely situations 
and faults is better decided at execution: the health monitoring 
system passes a belief over the system state to the executive 
that decides which portion of the plan to execute, sometimes 
inserting/replacing wanted/unwanted plan blocks. 

More recent architectures try to mitigate these problems by 
moving towards unified planning and execution frameworks 
(Alami et al 1998; Muscettola et al 2002; Estlin et al 2005^ 
Several of these architectures are discussed at the end of this 
paper, however it is well understood that uncertainty in future 
values forces an agent to plan locally. For example, to mitigate 
this problem, (Muscettola et al 2002) allows plans to include 
explicit calls to a deliberative planner. This comes back to 
finding place where to insert a branch, and as demonstrated 
in (Dearden et al 2003), the branch point is usually not sit- 
uated at the point that has the highest probability of failure. 
Now note that if the process of estimating a good branching 
point does not forcely require to do the planning, it doesn’t 
cost much to pre-plan the branch once the point has been iden- 
tified. Therefore, the branch can be pre-planned and its values 
later updated during execution. As it will be explained later 
in this paper, re-evaluation of a plan is in no way equivalent 
to re-planning, but a re-evaluated plan can be found that is op- 
timal w.r.t. the information on the uncertain system state and 
the original plan. 

We said that most planners do not handle well the complex- 
ity due to the presence of faults in a model and therefore rarely 
include faults within their planning domain. Moreover, major 
faults are well known and recoveries can be efficiently con- 

1 Partially Observable Markov Decision Processes (POMDPs) al- 
low the latter but are often untractable. 


structed before execution. At runtime, a fault detection sys- 
tem, or more generally, a state estimator will return a state 
estimate that triggers one or more plan fragments for system 
recovery or opportunistic science. These plan fragments are 
often referred to as floating contingencies whose execution 
can be conditionned upon resources (including time) and/or 
system behavioral modes. Therefore in this paper we will re- 
fer to two types of contingencies: pre-planned branches on 
resources that are part of the main plan, and floating contin- 
gencies, that trigger in response to certain events and resource 
values. The paper focuses on techniques to re-evaluate the for- 
mer, and studies the complexity added to them by the latter. 

The problem can be seen as one of re-evaluating the plan 
values, such as its utility, and updating the plan conditionals, 
i.e. the branch conditions. Typically, at runtime, the proba- 
bility mass of the state estimate shifts among regions of the 
hybrid space (continuous resources plus discrete state). We 
adapt the pre-computed branch conditions to these changes 
by projecting the changes forward and backing up the result- 
ing states. Our first approach is an adaptation of the clas- 
sical Monte Carlo (MC) technique (Sutton & Barto 1998; 
Thrun 2000). Our second approach is based on decision 
theoretic techniques and converts the problem into a small 
Partially Observed Markov Decision Problem (POMDP)(see 
(Kaelbling, Littman, & Cassandra 1998) for an introduction 
and more references) whose solving at runtime returns proba- 
bilistic decision lines that are optimal given the initial plan. 


Preliminaries 

Here a plan can be seen as a tree whose nodes are known as 
the branch points. The value function for a node is a continu- 
ous function over the multi-dimensional resource state, i.e. a 
mapping from the resource space to the utility space, and that 
depends on downstream node value functions. Planning de- 
termines a set ofpolicies that maximize the expected utility of 
the plan. At branch points, this leads to conditions over the 
resource space that discriminate among branches. 

Typically, planning proceeds to a mapping from the system 
state space to the utility space, i.e. the utility obtained by ex- 
ecuting the plan, that it seeks to maximize. Noting the system 
state s — (x,r) with x € X the discrete state (or system 
modes), and r £ R the multi-dimensional continuous state 
(including time), the utility earned by executing a branch 6* 
starting at s can be noted: 


v b, (s) = E / p((x',r') I *,Oii)[^(oii,(* , ,r / )) 

x'€X J R 

+ V Bi {x',r')}dr' ( 1 ) 

with a^i the first action of branch 6*, Bi the remaining portion 
of the branch, U(an> (x\ r')) the utility earned, and s' the 
system state after executing fyi following the probability dis- 
tribution p(s f | s,a ti ). Over a belief state 7 r(s), as estimated 
by the health monitoring system, we have: 


Vb^s)) = E / V bi (x,r)n(x,r)dr (2) 


And at a branch point where n branches are available, the best 
branch is decided according to: 

b* = arg max (?r(s)) (3) 

i€[l,nj 


This is similar to the Bellman equations for POMDPs (Boyan 
& Littman 2000). Each value function V ( b ) maps the resource 
space to the utility of the branch b. The max operator of re- 
lation (3) defines an upper bound on the branch point overall 
utility value, and branch conditions are found at the functions 
intersections. At execution, deviations from the planning do- 
main and information of the state estimate move these decision 
lines. 

There are several conditions and situations under which the 
plan value must be re-evaluated. First, when the execution 
encounters a branch point, any change in the Bellman equa- 
tion functions, such as the belief b over the state s, the reward 
model [/, the action cost model, requires that all branch func- 
tions at this branch point are re-evaluated. Second, if not at a 
branch point, but if a floating branch has to be inserted, then 
the plan equation is changed and the remaining portion of the 
currently executed branch as well all future branch conditions 
must be re-evaluated. For example, when inserting a branch 
6/, equation (1) becomes: 


V bf (s) = V bf (s) + E / *>((*'• r 0 I s,b } )V B {x'y)dr' 


(4) 


where B is the remaining portion of the current plan to be ex- 
ecuted after bf. The local value of bf is the expected reward 
from the actions within the floating branch itself. The remain- 
ing term is a representation of the end state of the local plan, 
including the probability of the resources remaining after exe- 
cuting the local plan. 

The remaining of the paper studies approaches to the fast 
re-evaluation of these decision lines. 


The Monte-Carlo Approach to the 
Re-Evaluation of Contingency Plans 
Approximating branch average utility 

Applying Monte Carlo techniques to the approximation of 
equation (2) is straightforward: the integral over the multi- 
dimensional continuous space is turned into a sum by sam- 
pling N times from b(s) and p(s' | s,a), and the utility is 
averaged over the successive runs. We note: 

= E (5) 

xtxx'ex 

where s' ~ p(s' j Sj,an) and s 3 ~ 6(s). The larger the 
N , the better the fit to the underlying probability distributions, 
and the better the approximation. 

Plan simulation 

For simulating branches with MC, we use a prioretized pile of 
events including plan actions, and a set of constraints among 
them. The pile is filled up with actions whose execution is 
simulated by testing their temporal constraints and sampling 
their consumption before being rewarded and popped out. 



Sampling decisions 

We sample the decision by deciding the path with highest util- 
ity for each sample. We write: 

1 N 

V dec (n(s)) = — .“ax, V* (*(*)) (6) 

iV * — * l,n] 

3=1 

In algorithm 1, each path is explored by each sample for the 


1: for all j < TV do 

2: Proceed with MC on the first branch. 

3: for all branches bi at branch point do 

4: Apply this algorithm recursively to b with j = 1. 

5: Return the highest utility at this branch point (max). 

6: Return the averaged utility of the plan. 

Algorithm 1: Recursive procedure for sampling decisions 

evaluation of the max operator. The averaged returned utility 
is near optimal, but the sampled decision for the best branch 
(the arg operator) depends on the sampled resource space that 
must be partitioned into subregions of identical decision. 

Floating contingencies 

Floating contingencies are a challenge to the simulator be- 
cause they can trigger at anytime. The simulator uses ran- 
dom events to trigger these branches and specific dynamic 
constraints to handle their insertion. The complexity increase 
due to floating branches is a product of the number of plan ac- 
tions, actions in the branch, and the number of these branches. 
The next section covers the retrieval of the decision lines in 
the multi-dimensional resource space. 

Bounding the resource space for deciding future 
branches 

Decision at branch points can be made based on the simula- 
tion results by executing the branch with the highest earned 
utility average. Simulation provides sufficient information for 
computing branch conditions at future branch points. This op- 
eration is performed at virtually no cost and can spare future 
simulations by constraining future decisions. 

Approximating branch decision lines thru piecewise con- 
stant value function approximation Our solution is to slice 
the resource domain into rectangular bins and to fit the branch 
value functions in each bin with a piecewise constant function, 
based on the MC samples. Function intersections are found at 
bin edges. Noting A r a bin in the resource space, we can write 
bi's value: 

K (*(*)) = E E P ^ I Ar) P (A T )V bi (A r , X) (7) 

xex 

i.e. as the sum of the average utilities of bi in each bin when it 
is the branch with the highest expected utility. More precisely: 

V bi (A r ,x) = — E E^fcfo) (8) 

rA - rj €A r x'eX 


with s = (z,r) and sj — ( x* ,rj ), is the average utility of bi 
on bin A r from the n rAr samples r 3 it contains, 

p(bi | A r ) = -2- E = ar 9 max V^Wsj)) (9) * 

ry£A r * €|1 ’ n| 

where 5 is the Dirac function, is the probability for bi to be the 
branch with the highest utility over the samples of the bin, 

p(A r ) = (10) 

is the probability of the bin itself. An optimal bin size W is 


1: Proceed with algorithm 1 and collect samples at branch 
point. 

2: for all branch points in the contingency plan do 
3: Compute the optimal bin size and slice the space into 

bins. 

4: Compute statistics with equations (8), (9) and (10). 

5: Evaluate equation (7) for each branch. 

6: In each bin, identify the branch with the highest value. 

7: Identify new branch conditions where successive bins 

have different highest utility branches. 

Algorithm 2: Branch conditions approximation thru piece- 
wise constant value function approximation 

obtained, in the sense that it provides the most efficient unbi- 
ased estimation of the probability distribution function formed 
by the samples. We used W = 3.49criV -1 / 3 where a is the 
standard deviation of the distribution, here estimated from the 
samples (D. 1976; A.J. 1991). The overall strategy is pre- 
sented on algorithm 2. 

Branch conditions are obtained by comparing the branch 
with the highest utility for each bin: if two successive bins re- 
turn different results, a branch condition exists at their edge. 
Thus, the precision of the approximation is directly depen- 
dent on the optimal bin size, that depends on the number of 
samples. Stutter at decision point can be overcome by fitting 
the successive piecewise constant approximations with more 
smoothly curve. 

Belief update on re-evaluated branch conditions The re- 
evaluated decision functions are inequalities of the form r' < 
(>)g(r). Given a state estimate i r(s) at branch point, decision 
over n branches follows: 

b * = arg max / Vb i {x,r)'ir(x^r)dr 

X ex^ r ^(r) 

« arg max E E p ^» I Ar ' )P(^r')Vb(A' r , x)ir(r(}±) 

with r l such that Vr' E A r /, r f < g(r). 

Discussion 

The major drawback of the Monte-Carlo approach is that it 
provides a probabilistic guarantee of its results, that is never 
absolute. This is a problem that we partially address in the next 




section with the use of a decision theoretic formulation. An- 
other work, (Jain & Varaiya 2004) finds bounds on the number 
of samples for the convergence of the expected reward for a 
class of policies. 

Decision theoretic approach to plan 
re-evaluation 

Another problem with the MC approach is that the decision is 
made based on a mapping from the continuous resource space 
to the utility space that forces the approximation of the deci- 
sion lines. An alternative is to use a mapping from the be- 
lief space over the decisions to the utility space. The decision 
space is finite, made of the branch conditions of the original 
plan. The belief space over the decision is continuous and of 
dimension the number of decisions minus one. This formula- 
tion leads to an enlarged space but allows the use of decision 
theoretic techniques to directly incorporate the belief space in 
the computation of optimal decision lines. More precisely our 
problem can now be casted into a small POMDP whose ac- 
tions are the plan branches, the states the branch conditions, 
the observations the system states. 

Plan reduction to a POMDP 


where in the absence of floating contingencies (because / can 
only lead to b ): 

P(f I b,f) = P(f | b)=J2 P(f I b, s')p(s') (15) 

s'es 

and R(f,b,s) = V b (b(s)), from equation (2). Finally the 
value of executing branch b from some belief state ir and ob- 
serving s is: 

V.(7Tfr)=5> (f,s)V(f,s) (16) 

/GF 

and the optimal value function is given by: 

V(n) = m^y^p(s)K s (7r 6 ) (17) 

s€S 

Simulation 

The successor states s' and and the p(s') of equation (15) are 
unknown and must be obtained through simulation. As a simu- 
lator we use the MC algorithm of the previous section and gen- 
erate both the V b {s) and the s' in a depth first forward search 
in the plan tree. 


A standard POMDP is made of a set of actions, a set of states, 
a set of transitions among states per action, and a set of obser- 
vations. In our model, we abstract away the actions and use a 
branch an action for the POMDP. Our POMDP is then defined 
as a tuple ( F , 5, B.L , T, R) where: 

• F is a finite set of branch decision outcomes (as states), 

• 5 is a finite set of system states (as observations), 

• B is finite set of branches (as actions), 

• P(s | 6, /') is the probability of state s given that branch b 
has been executed and has landed in /', 

• P(f f I £>?/) is the probability of entering outcome /' after 
taking branch b in outcome /, 

• i?(/, b) is the reward for taking branch b while in outcome 
/■ 

The POMDP belief update can be expressed as: 


Kb(f,s) 


p(s | M) 


( 12 ) 


where tt is a probability distribution (belief) over F , given s 
and b, and: 


P( ,i U1 = WM<w us) 

P(f ) 

The value of executing branch b under decision / and state s 
is: 


V(f,s) = R(f,b,s) 

+ 7 £ P(f I Pi** I /'> b)V(s\ /') (14) 

f'€F s‘€S 


Solving 

The solving of this POMDP returns a piecewise linear convex 
value function that is a mapping from the belief space over the 
decision outcomes to the highest expected plan uiiity. Optimal 
branch conditions are found at the intersections of maximized 
value functions and are now conjunctions of inequalities of 
the form P(r < h(r)) < c where r < h(r) is the branch 
condition from the original plan and c a constan in [0, 1]. For 
any belief over an outcome, the solution returns the optimal 
policy, w.r.t. the original plan. 

Floating contingencies 

Floating contingencies pose a serious problem to the decision 
theoretic approach because the possible interruption of any 
action within a branch leads to a potentially infinite number 
of actions (breaking up a branch an infinite number of times 
over resource and time values with non null probability). Ap- 
proaches like (Younes & Simmons 2004) can be used here 
to handle the asynchronous events, but do not allow to in- 
clude the events (here floating contingencies) within the policy 
(therefore the computation of their conditions is not possible). 
While we are not yet sure about the range of solutions to this 
problem, it seems realistic to research approximations of float- 
ing conditions over a single branch. 

Results 

A contingency plan for the Mars exploration domain 

Our application is on a planetary rover plan. Consider the plan 
for a Mars rover on figure 1. It tells the rover to first navigate 
to a waypoint w Q , and there to decide whether to take a high 
resolution image of the point (HI res) or to move forward to a 
second waypoint depending on the level of resources (here 
energy and time). After reaching and digging in the soil, it 
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(a) Contingency plan for the Mars rover domain 



(b) Value functions of branches at branch 
point 1 (bptl) 


(c) Value functions of branches at branch 
point 2 (bpt2) 


Figure 1 : Branch value functions at branch point for a detailed rover problem 


must decide whether to move forward to waypoints 103 or W 2 
or to simply get an image at w\ and wait for further instruc- 
tions. NIR is a spectral image of a site or rock. Action time 
and energy consumptions are represented as Gaussian bumps 
of empirical mean and variance. In this example branch con- 
ditions at branch points bptl and bpt2 have the following pa- 
rameters: ai = 0 . 1 , a 2 = 2.1 and /3 2 = 2 . 2 . 

Decision sampling 

Branch conditions re-evaluation at branch points: bounds/bins 
are generated with the sampling decision algorithm, and veri- 
fied by running a classical Monte-Carlo simulation, that does 
not maximize the titility, but follows the new branch condi- 
tions and averages the earned utility. Simulation also returns 
the failure probability of the plan. The error is the difference to 
the optimal plan value in percentage. The piecewise constant 
approximation of the branch value functions returns godd util- 
ity (Table 1 ). Figure 2 pictures results for the second branch 
point of our rover problem (the energy is pictured and the time 
line is omitted) and shows the shifting branch conditions on 
the horizontal axis that is the energy line. 


N 

Value 

Time 

V dec 

err 

100 

14.21 

0.03 

10.9 

23.3 

500 

13.618 

0.16 

9.732 

28.53 

2500 

13.8244 

0.78 

11.2992 

18.26 

12500 

13.8008 

4.08 

11.9542 

13.27 

62500 

13.7835 

20.79 

12.156 

11.8 

312500 

13.7717 

120.3 

12.1214 

12 

500000 

13.777 

223.89 

12.1814 

11.58 


Table 1: Monte-Carlo decision sampling and branch condition 
re-evaluation based on MC samples. Results are as follows: 
N is the number of samples, V is the mean expected highest 
value obtained for the plan, V dec is the value obtained when 
using the re-evaluated branch conditions, err the error per- 
centage to the simulated best value. 
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P(pi |Ar)Vf> i (A r ) 


Figure 2 : Piecewise constant approximation of branch value functions from simulation samples. 



Figure 3 : Optimal value function over the original plan branch 
conditions: x and y axis represent the probability of decision 
outcomes P(r < 0:2) and P(a 2 > r < /? 2 ). P(r > / 3 2 ) is 
deduced from them. 


Decision theoretic approach 

We converted our example to a POMDP and simulated the 
observations and rewards, respectively the system states and 
branch value functions. Starting from a fixed level of re- 
sources, figure 3 shows the convex value function solution for 
the second branch point (&pt 2 )- 

Comparison and Discussion 

To compare the two approaches, we moved a gaussian belief 
of fixed variance 0.1 along the resource (energy) line and stud- 
ied the decision for each resource value. Results are presented 
on figure 4 . V me and dec me respectively denote the value 
obtained and the decision based on the Monte-Carlo method 
with b 5 = 1 , &4 = 2 and b 3 = 3 ; V dtp and dec dtp are based 
on the decision theoretic planning (dtp) approach. First, the 
difference in value between the two methods is due to the high 
level of branch failure (i.e. resource gets to zero) in the sim- 
ulation for the decision theoretic approach (since it is based 
on the original branch conditions). This is of medium impor- 
tance only when we study the decision making: we observe 



Figure 4 : Comparition between the Monte-Carlo and the deci- 
sion theoretic re-evaluation methods. 

that decision to go to branch 64 then to b 3 based on MC are 
respectively slightly early and slightly late, and this is visu- 
alized as two sudden drops in expected value; the decision 
based on dtp switches later to £>4, and earlier to 65 right at 
the highest value point. Overall the dtp-based decision leaves 
less room for branch 64, which can be surprising when look- 
ing at the large surface corresponding to 64 on figure 3 but is 
explained by the fact that a high probability for on decision 
2.1 < r < 2.2 denotes a more accurate belief (given the fixed 
variance) than for other branches. Finally it is difficult to fully 
assess the dominance of one method over the other. At this 
point of our research we lean in favor of the MC approach 
for plans with a small number of actions and a high number 
of decision outcomes, and for the dtp approach when a high 
number of actions in a plan makes the price of successive MC 
simulations costly. 

Related Work and Conclusion 

We have presented a simple strategy for the robust execution 
of contingency plans under uncertainty. It re-evaluates branch 
value functions at branch point and re-estimates branch con- 
ditions whenever necessary. The framework allows runtime 
insertion/replacement of plan portion thru the use of floating 



branches but much work on their full integration into the re- 
evaluation process should follow. This is the first step towards 
the development of more powerful techniques for planning 
and execution under uncertainty. The MC approach is flex- 
ible and provides good results in any situation given that a 
sufficiently high number of samples is used. The algorithms 
presented are a baseline capability, and will be used later to 
assess the quality of more complex and focused approaches. 

Related Work 

Other works on plan re-evaluation include (Gough, Fox, & 
Long 2004) that studies plan execution with uncertainty on 
the resource consumption. However, the executed plans are 
no contingency plans as branch execution is not conditionned 
on decision functions over the resource state. (Washington 
& Lees 2004) develops a fast method for plan portions in- 
sertion/replacement, but partly fails here as it relies on pre- 
computed value functions (this is not always possible as faults 
change the model of actions). 

Mixed planning/execution include (Alami et al. 1998) that 
uses a deliberative planner and an executive on top of a set of 
reactive controllers. (Estlin et al. 2005) presents the Closed- 
Loop Execution and Recovery (CLEaR) system that is in- 
tended to run on rovers with little communication with ground. 
CLEaR closely integrates the CASPER continuous planner 
(Chien et al. 2000) and the TDL executive system (Simmons 
& Apfelbaum 1998). Plan re-evaluation and the methods de- 
scribed in this paper can be seen as an alternative to the iter- 
ative plan repair of CASPER. We view plan re-evaluation as 
an intermediate step between execution of pre-planned con- 
tingencies and re-planning. Re-planning will always be neces- 
sary as if a situation occurs on-board for which there is no pre- 
planned contingency, the rover must wait for instructions. In 
that sense, plan re-evaluation complements architectures such 
as (Muscettola et al. 2002) and (Estlin et al. 2005). 

For solving the decision theoretic problem, fast techniques 
such as (Feng & Zilberstein 2004) allow the solving of rather 
large problems. Given we abstract away actions within the 
branches when formulating the POMDP, we see our problems 
(not including the floating contingencies) as being of a small 
size. 

Future Work 

Future work includes dealing with floating contingencies 
within the decision theoretic framework, pre-computing more 
advanced branch value functions at planning time (Feng, 
Meuleau, & Washington 2004) and using them at runtime. An- 
other hot topic remains the re-evaluation of plans that contain 
concurrent actions. Also note tha new class of problems re- 
cently arised in the rover domain, where th$ robot is able to 
satisfy only a subset of the goals (Smith 2004). In that case, 
re-evaluating the plan is not as efficient anymore because the 
change in resource consumption would in general lead to the 
selection of a different subset of goals. 
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